Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apache spark images automated build pipeline #1

Merged
merged 17 commits into from
Apr 4, 2024
Merged

Apache spark images automated build pipeline #1

merged 17 commits into from
Apr 4, 2024

Conversation

idirze
Copy link
Contributor

@idirze idirze commented Apr 4, 2024

The first version contains:

  • spark-base, spark (java/scala), spark-r (minimal R requirements), spark-py (minimal python requirements).
  • The images except spark-base come with the following okdp extensions:
  • The size of the spark image is less than 450MB (minio/aws S3 dependencies minimized)
  • Automated build, test (CI), push and release pipeline.
  • Automatic weekly rebuild of the images (security updates)

idirze added 17 commits March 28, 2024 11:40
@idirze
feat(spark-base): Add new spark-base image (java/scala only) without …
59eb7d8 
…okdp extensions
@idirze
feat(spark): Add new spark image (java/scala only) with okdp extensio…
1d935f7 
…ns (aws/minio, prometheus java agent, okdp-spark-auth-filter)
@idirze
feat(spark-py): Add new spark-py image with python basic requirements
ad94e07
@idirze
feat(spark-r): Add new spark-r image with R basic requirements
a9c0880
@idirze
Build pipeline - Add basic pipeline to build and push into CI registry
2201f45
@idirze
Build pipeline - Add scala 2.12/2.13, python and r integration tests …
cbabf51 
…for images (spark > 3.3.0)
@idirze
Build pipeline - Publish to official registry
22ffda2
@idirze
Build pipeline - Publish to official registry repo based on the lates…
fc9d948 
…t Github release/pre-release tag
@idirze
Spark image - Remove Control M characters from spark parent pom.xml
c76825c
@idirze
Build pipeline - Add action to run CI and publish against versions an…
03828a4 
…d reference versions matrices
@idirze
Build pipeline - Add branch name as suffix for CI images latest tag
f0e6088
@idirze
release-please - automate release/publish process and periodic images…
fd92972 
… rebuild
@idirze
fix(spark-base): Add missing gpg keys in the spark project release keys
980b001
@idirze
Add license file
d81faae
@idirze
Update documentation
8f55245
@idirze
feat(spark): Minimize minio/aws sdk v1/v2 depedendencies to reduce sp…
5da5b8a 
…ark image size
@idirze
Prepare first official release
4796814
@idirze idirze merged commit 19e70be into main Apr 4, 2024
43 checks passed
@idirze idirze deleted the build-pipeline branch April 4, 2024 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mlahouar mlahouar mlahouar approved these changes

@SergeAlexandre SergeAlexandre SergeAlexandre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@idirze @SergeAlexandre @mlahouar